Q&A

./install.sh`
Git LFS initialized.

git lfs install
Git LFS initialized.

git lfs track "*.psd"`
Adding path *.psd

git add path/to/file.psd

git commit -m "add file.psd"
git push

Sending file.psd
44.74 MB / 81.04 MB  55.21 % 14s
64.74 MB / 81.04 MB  79.21 % 3s

Why To Write Unit Tests

Testing is essential, testing is hopeless. If you are changing code that works, fixing code that doesn’t, or writing new code, you need a way to tell when it isn’t broken. Computers and the demands placed on them by software applications are now so complicated that any hope of testing them to saturation is lost. You cannot wait long enough for every possible input to each permutation of your code execution path to be analyzed. Without unit testing, you will not be able to prove that a piece of code does what it is intended to do, unless you exercise the entire system under realistic conditions - the malady a developer named Lindsay Page once referred to as “driving the car to see if the radio works.” So if you need to prove that a piece of code works, and you don’t have infinite time and money to devote to doing that, you need unit tests. You also need to write unit tests to crystallize something essential which is sometimes separate from the code - the developer’s intent.

It’s your job as the software engineer to exercise judgment, dicing the code into manageable units and distilling as many of the most important cases you can think of into a set of inputs and the expected set of outputs from those units.

What You Need

1. Expected inputs
2. Expected outputs
3. Some code
4. A way to run 3. and provide 1. to check the results against 2.

If you don’t have 1. or 2., you are working on a piece of software with no requirements. If you are working on generative art, this is OK. If you are not, you need to show stakeholders (people with money or risk involved in your project) that your software does what they want, and doesn’t do what they don’t want.

Having specific requirements that change on timescales longer than your workday is essential to writing tests, and it may be a significant amount of work to get them from stakeholders so you can begin writing tests.

If you don’t have 3., don’t get any, otherwise you will have to write unit tests.

If you don’t have 4., you are in the right document.

How To Do It

The process of unit testing is: isolate a chunk of code as much as possible, write code to compare the output to an expected value, run it, do this many times, and collect the results.

To isolate the unit under test (UUT), we will write code that #includes the code to test, or links to a compiled library. As necessary, we will lift dependencies by introducing stub functions for them or overwriting them with our own, simpler ones.

To write the test code, we will leverage a unit testing framework that provides macros to make checking for agreement between returned values and expected values quick and easy.

To run it, we will add the test code to our build system, and make sure the test executable runs as a part of the build process.

To collect the results, we will again leverage the unit testing framework to generate a test report.

When To Do It

Entire books have been written about this topic, so I will try to simplify by suggesting when more tests should be written for the your big ugly legacy project, given that it is ostensibly already-working, field-tested code. Sometimes, the most cost-effective thing to do is nothing if the code works, but if any of these are true:

• You are about to write a new module
• You are about to troubleshoot an old module
• Commits repeatedly result in regressions
• Flight code exhibits segmentation faults, race conditions, or memory leaks and workarounds are observed instead of root cause fixes
• More than 60% of dev time is spent debugging (heuristic, your mileage may vary)

then time needs be allocated to developing unit tests in the affected modules.

Choosing a Unit Test Framework

There are many more ways to write unit tests than there are programming languages, so rationales for choosing one test framework over the other like personal preference, organizational inertia, and random chance are all good ones. The most important aspects of your chosen test framework to focus on are, in order:

1. Having one
2. Speed: testing overhead must be small, so developers will want to run them early and often as part of their development routine
3. Extensible: it must be easy to add new tests, so developers will want to add new tests for their new features, and maintainers will want to add tests for legacy untested features

The goal of this document is to describe a method for testing C/C++ code that accomplishes 1., and arguments can be made that it accomplishes 2. and 3. For C/C++ code, some of the most popular frameworks are Google Test, Catch, CppUnit, Boost.Test, and nothing. It would be nice to use Google Test for the TIM project because it’s well documented and the chances are high that we will see it again if we continue to work on C/C++ projects. However, it introduces a C++11 dependency, and the cleanest and easiest way to install it and maintain it on our target system is a feature added in a more recent version of CMake (3.11) than we have access to (<= 3.0.5).

There are other test frameworks geared more toward C, and for this example we will use a lightweight one (2 headers, 1 source file) called Unity. This is advantageous for our project because it allows the flight software to continue to be self contained, and should not rely on fetching any code from the internet at build-time. Unity also has more advanced companion tools that may become useful if our testing needs grow.

Finally, learning any unit test framework is like learning a language, in that it will have benefits even if you go on to code in something other than C.

Unity

Unity is made by ThrowTheSwitch.org.

There is a useful Getting Started guide.

A quick and dirty way to make the Unity source available could be embedding itin the repo so it can be found by more than one executable.

There is a tradeoff here: embedding a copy of the Unity source makes it more difficult to accept updates to it from upstream, but it makes writing CMakeLists.txt files easier. In this example, I set aside my usual preference to have dependencies managed separate from flight code because I think writing CMake will happen more often than pulling Unity updates.

evanmayer@framework:~/unittest_example$ tree -L 2
.
├── CMakeLists.txt
├── external
│   ├── CMakeLists.txt
│   └── Unity-2.5.2
└── src
    ├── add_example.c
    ├── add_example.h
    ├── CMakeLists.txt
    └── test_add.c

#include <unity.h>
#include "add_example.h"

void setUp(void) {
    // set stuff up here, if needed
}

void tearDown(void) {
    // clean stuff up here, if needed
}

void test_AddOnePlusOne(void)
{
    int a = 1;
    int b = 1;
    TEST_ASSERT_EQUAL(2, add_example(a, b));
}

int main(void) {
    UNITY_BEGIN();
    RUN_TEST(test_AddOnePlusOne);
    // append additional tests here
    return UNITY_END();
}

cmake -S . -B build
cmake --build build
cd build/ && ctest && cd ..

Test project /home/evanmayer/unittest_example/build
    Start 1: add_example_test
1/1 Test #1: add_example_test .................   Passed    0.00 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.00 sec

What is “real-time”?

Software for aerospace and industrial systems has different requirements than most software people regularly interact with. It may have stringent deadlines to meet when interacting with the real world or other systems which, if not met, result in behavior that could endanger the mission goals, the system itself, or things around the system (such as people or property).

For example: an electronic coffee machine heats water from a starting temperature to one set by the user. To heat the coffee, it energizes a heating element. To determine the present temperature, it reads a thermometer. In order to stop heating the water at the appropriate time to avoid overheating and boiling off all the water, it must measure the temperature at a given rate, perform a calculation (like the difference between the present temperature and the set point), determine whether the heater should be on or off, and command the heater if necessary. What happens if any one of these steps is delayed? If the temperature probe does not return a valid measurement for several seconds for some reason, and the rest of the system does not have any means to compensate for missing measurements, the temperature may miss the set point, resulting in poor temperature control or excess water boil-off.

The consequences of any mis-timing in even this simple system depend on the frequency and magnitude of the mis-timing event. If a temperature query or heater on/off command deadline is intended to happen every 100 Hz, but is delayed one time by a dozen milliseconds in a 3-minute run of the coffee machine, the end user will not notice and the coffee quality or boiler safety will not be impacted. However, if it happens more often or with greater severity, the chance of failure goes up!

This simple illustration introduces a few principles of real-time systems that impact their design and testing:

• The system needs to be fast enough to minimize the severity of exceptional circumstances where a deadline is not met
• It also needs to be stable enough to minimize the number of these occurrences

In my opinion, there is no black and white “real-time,” only “fast enough” and “stable enough” for the application’s requirements! The coffee machine controller may be fine to heat a boiler of hot water, but risks failing spectacularly when it is required to control cooling of a pulsed laser.

Linux RT FAQ: What is real-time?

Linux real time

Running software for a complex vehicle on a PC with a Linux operating system turns the problems illustrated above up to eleven. There are many sensors and actuators to interact with the outside world, and the operating system itself has various tasks to run, which may not be on a predictable cadence. The resource needs of the system fluctuate. In order to deal with this fluctuation and the reality that there may sometimes be insufficient resources (CPU cycles vs. time, memory) to perform the necessary tasks, we must carefully set up our system.

If the system is not able to completely eliminate all events that would be severe enough to cause a failure, it must have safeguards in place to deal with them, such as giving up on a blocking process and re-using an older sensor value, or predicting a new one from past data.

The most important point is that ALL of the measures outlined below work together to limit the frequency and severity of events that “break real time.”

NOTE: This is a good (but evolving) resource on it: https://rt.wiki.kernel.org/index.php/Main_Page

More info:

• https://lwn.net/Articles/146861/
• https://rt.wiki.kernel.org/index.php/Main_Page
• https://elinux.org/images/d/de/Real_Time_Linux_Scheduling_Performance_Comparison.pdf
• https://rt-labs.com/docs/p-net/linuxtiming.html
• https://chenna.me/blog/2020/02/23/how-to-setup-preempt-rt-on-ubuntu-18-04/
• https://wiki.linuxfoundation.org/realtime/documentation/howto/tools/cyclictest/start
• http://www.cs.ru.nl/J.Hooman/DES/XenomaiExercises/Background.html
• https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.55.2137&rep=rep1&type=pdf